Remove CFFI in favor of native extension to support free-threaded mode #96

adriencaccia · 2025-10-28T15:19:59Z

Fix #86

codspeed-hq · 2025-10-28T15:37:19Z

CodSpeed Performance Report

Merging #96 will degrade performances by 24.69%

_{Comparing feat/remove-cffi (411adc2) with master (adee8a1)}

Summary

⚡ 47 improvements
❌ 22 regressions
✅ 98 untouched

⚠️ Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

	Mode	Benchmark	`BASE`	`HEAD`	Change
⚡	Simulation	`test_iir_filter_process`	35.2 µs	32.8 µs	+7.29%
❌	WallTime	`test_iir_filter_process`	2.6 µs	2.7 µs	-3.85%
⚡	Simulation	`test_iir_filter_set_coefficients[a_coeffs0-b_coeffs0]`	16.7 µs	14 µs	+19.03%
⚡	Simulation	`test_make_allpass`	46.4 µs	43.8 µs	+5.99%
⚡	Simulation	`test_make_bandpass`	47.1 µs	44.5 µs	+5.93%
⚡	Simulation	`test_make_highpass`	47.2 µs	44.5 µs	+6.13%
⚡	Simulation	`test_make_highshelf`	46.1 µs	43.6 µs	+5.72%
⚡	Simulation	`test_make_lowpass`	47 µs	44.5 µs	+5.81%
❌	WallTime	`test_make_lowpass`	5.3 µs	5.5 µs	-2.81%
⚡	Simulation	`test_make_lowshelf`	46.1 µs	43.6 µs	+5.8%
⚡	WallTime	`test_make_lowshelf`	7.4 µs	7.2 µs	+3.3%
❌	WallTime	`test_make_peak`	6 µs	6.2 µs	-2.73%
⚡	Simulation	`test_make_peak`	43.3 µs	40.9 µs	+5.9%
⚡	Simulation	`test_color[graph0-3]`	76.7 µs	74 µs	+3.68%
⚡	Simulation	`test_combination_lists[0-0]`	26.5 µs	23.9 µs	+10.78%
⚡	Simulation	`test_combination_lists[4-2]`	31.3 µs	28.7 µs	+8.81%
⚡	Simulation	`test_combination_lists[5-4]`	31.4 µs	28.9 µs	+8.65%
❌	WallTime	`test_combination_sum[candidates0-8]`	10.9 µs	11.1 µs	-2.62%
⚡	Simulation	`test_combination_sum[candidates0-8]`	43.6 µs	40.9 µs	+6.61%
⚡	Simulation	`test_depth_first_search[4]`	86.2 µs	84.2 µs	+2.34%
...	...	...	...	...	...

ℹ️ Only the first 20 benchmarks are displayed. Go to the app to view all benchmarks.

Vizonex · 2025-10-29T21:56:39Z

Fix #86

This is still a work in progress, it must be tested thoroughly in CI to ensure that it plays well with free-threaded versions of Python, using UV or not.

Sounds good. I can't wait to see if this will pass on through or not.

adriencaccia · 2025-10-30T09:00:46Z

Sounds good. I can't wait to see if this will pass on through or not.

It seems that there is an issue when running CODSPEED_LOG=debug codspeed --mode instrumentation -- uv run pytest tests/ --codspeed with python3.14t
Here is the error:

Fatal Python error:   ⠠ Running the benchmarks...                                                                                                                                                                                                                                                                      
preconfig_init_allocator:   ⠠ Running the benchmarks...                                                                                                                                                                                                                                                                
PYTHONMALLOC: unknown allocator
Python runtime state:   ⠠ Running the benchmarks...                                                                                                                                                                                                                                                                    
preinitializing  ⠠ Running the benchmarks...                                                                                                                                                                                                                                                                           


Stack (most recent call first):
  <tstate is freed>

In the runner we are setting the following:
https://github.com/CodSpeedHQ/runner/blob/e89b00befed092f3300c89b9f6577a00e025c944/src/run/runner/valgrind/measure.rs#L91

Do you know if there is any issue with free-threaded python and using PYTHONMALLOC?

Vizonex · 2025-10-30T16:57:45Z

Sounds good. I can't wait to see if this will pass on through or not.

It seems that there is an issue when running CODSPEED_LOG=debug codspeed --mode instrumentation -- uv run pytest tests/ --codspeed with python3.14t Here is the error:

Fatal Python error:   ⠠ Running the benchmarks...                                                                                                                                                                                                                                                                      
preconfig_init_allocator:   ⠠ Running the benchmarks...                                                                                                                                                                                                                                                                
PYTHONMALLOC: unknown allocator
Python runtime state:   ⠠ Running the benchmarks...                                                                                                                                                                                                                                                                    
preinitializing  ⠠ Running the benchmarks...                                                                                                                                                                                                                                                                           


Stack (most recent call first):
  <tstate is freed>

In the runner we are setting the following: https://github.com/CodSpeedHQ/runner/blob/e89b00befed092f3300c89b9f6577a00e025c944/src/run/runner/valgrind/measure.rs#L91

Do you know if there is any issue with free-threaded python and using PYTHONMALLOC?

@adriencaccia Sorry for my rather late response but there is one called PyMem_RawMalloc which can be ran without the gil same with PyMem_RawFree, PyMem_RawRealloc, PyMem_RawCalloc

…adata Fixes #86

Removed cffi package version 1.17.1 and its dependencies from the lock file.

Copilot

Pull request overview

This PR migrates from CFFI-based Python bindings to native CPython extension module to enable free-threaded Python support. The change removes the CFFI dependency and replaces it with a hand-written C extension module that wraps the existing instrument hooks library.

Key changes:

Removed CFFI dependency and replaced with native C extension API
Added new C module instrument_hooks_module.c implementing Python bindings using CPython's C API
Updated InstrumentHooks class to use the new native module instead of CFFI's lib object
Modified callgrind function calls to be methods on InstrumentHooks instead of lib properties

Reviewed changes

Copilot reviewed 7 out of 8 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/pytest_codspeed/instruments/valgrind.py	Updated callgrind instrumentation calls to use instance methods instead of lib object
src/pytest_codspeed/instruments/hooks/instrument_hooks_module.c	New native C extension module implementing Python bindings for instrument hooks
src/pytest_codspeed/instruments/hooks/build.py	Removed CFFI build configuration file
src/pytest_codspeed/instruments/hooks/init.py	Refactored to use native extension module and added callgrind methods
setup.py	Replaced CFFI extension with native Extension setup
pyproject.toml	Removed CFFI dependency and updated package data configuration
.github/workflows/ci.yml	Added Python 3.14t (free-threaded) to test matrix

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-28T11:25:20Z

src/pytest_codspeed/instruments/hooks/instrument_hooks_module.c

+    int32_t pid;
+    const char *uri;
+
+    if (!PyArg_ParseTuple(args, "Oiy", &capsule, &pid, &uri)) {


The format string 'Oiy' is incorrect. The 'y' format expects const char* with length, but based on the C function signature instrument_hooks_set_executed_benchmark(hooks, pid, uri) which takes const char *uri, this should be 's' for a null-terminated string instead of 'y' which is for bytes with length.

Suggested change

if (!PyArg_ParseTuple(args, "Oiy", &capsule, &pid, &uri)) {

if (!PyArg_ParseTuple(args, "Ois", &capsule, &pid, &uri)) {

Copilot · 2025-11-28T11:25:20Z

src/pytest_codspeed/instruments/hooks/instrument_hooks_module.c

+    const char *name;
+    const char *version;
+
+    if (!PyArg_ParseTuple(args, "Oyy", &capsule, &name, &version)) {


The format string 'Oyy' is incorrect. Similar to the previous issue, 'y' expects bytes with length. Based on the C function signature which takes const char* parameters, this should be 'Oss' for null-terminated strings.

Suggested change

if (!PyArg_ParseTuple(args, "Oyy", &capsule, &name, &version)) {

if (!PyArg_ParseTuple(args, "Oss", &capsule, &name, &version)) {

Copilot · 2025-11-28T11:25:21Z

src/pytest_codspeed/instruments/hooks/__init__.py

+        # Don't manually deinit - let the capsule destructor handle it
+        pass


The __del__ method with only a pass statement and comment should be removed entirely. An empty __del__ method can prevent proper garbage collection and the capsule destructor will be called automatically without defining __del__.

adriencaccia mentioned this pull request Oct 28, 2025

Stop pulling unused cffi as a direct dependency in core packaging metadata #93

Open

webknjaz and others added 4 commits November 27, 2025 16:35

Stop pulling unused cffi as a direct dependency in core packaging met…

362aca3

…adata Fixes #86

Remove cffi package from uv.lock

ce22c14

Removed cffi package version 1.17.1 and its dependencies from the lock file.

feat: remove cffi in favor of native extension

0c63536

chore(ci): add 3.14.t in test suite

411adc2

adriencaccia force-pushed the feat/remove-cffi branch from 33c1bfa to 411adc2 Compare November 27, 2025 16:36

adriencaccia mentioned this pull request Nov 28, 2025

feat(run): support simulation for free-threaded python CodSpeedHQ/runner#167

Merged

adriencaccia marked this pull request as ready for review November 28, 2025 11:24

adriencaccia requested a review from Copilot November 28, 2025 11:24

Copilot AI reviewed Nov 28, 2025

View reviewed changes

adriencaccia requested a review from not-matthias November 28, 2025 16:43

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Remove CFFI in favor of native extension to support free-threaded mode #96

Remove CFFI in favor of native extension to support free-threaded mode #96

Uh oh!

adriencaccia commented Oct 28, 2025 •

edited

Loading

Uh oh!

codspeed-hq bot commented Oct 28, 2025 •

edited

Loading

Uh oh!

Vizonex commented Oct 29, 2025

Uh oh!

adriencaccia commented Oct 30, 2025

Uh oh!

Vizonex commented Oct 30, 2025

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Nov 28, 2025

Uh oh!

Copilot AI Nov 28, 2025

Uh oh!

Copilot AI Nov 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	if (!PyArg_ParseTuple(args, "Oiy", &capsule, &pid, &uri)) {
	if (!PyArg_ParseTuple(args, "Ois", &capsule, &pid, &uri)) {

	if (!PyArg_ParseTuple(args, "Oyy", &capsule, &name, &version)) {
	if (!PyArg_ParseTuple(args, "Oss", &capsule, &name, &version)) {

		# Don't manually deinit - let the capsule destructor handle it
		pass

Remove CFFI in favor of native extension to support free-threaded mode #96

Are you sure you want to change the base?

Remove CFFI in favor of native extension to support free-threaded mode #96

Uh oh!

Conversation

adriencaccia commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codspeed-hq bot commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CodSpeed Performance Report

Merging #96 will degrade performances by 24.69%

Summary

Benchmarks breakdown

Uh oh!

Vizonex commented Oct 29, 2025

Uh oh!

adriencaccia commented Oct 30, 2025

Uh oh!

Vizonex commented Oct 30, 2025

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Copilot AI Nov 28, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

adriencaccia commented Oct 28, 2025 •

edited

Loading

codspeed-hq bot commented Oct 28, 2025 •

edited

Loading